Bi-calibration Networks for Weakly-Supervised Video Representation Learning

The leverage of large volumes web videos paired with the query (short phrase for searching video) or surrounding text (long textual description, e.g., video title) offers an economic and extensible alternative to supervised representation learning. Nevertheless, modeling such weakly visual-textual connection is not trivial due polysemy (i.e., many possible meanings a query) isomorphism same syntactic structure different text). In this paper, we introduce new design mutual calibration between achieve more reliable supervision Specifically, present Bi-Calibration Networks (BCN) that novelly couples two calibrations learn correction from vice versa. Technically, BCN executes clustering on all titles searched by identical takes centroid each cluster as prototype. All queries constitute set. learning then formulated classification over prototypes queries, text-to-query query-to-text calibrations. A selection scheme also devised balance Two large-scale datasets title, named YOVO-3M YOVO-10M, are newly collected weakly-supervised feature features ResNet backbone learnt (3M YouTube videos) obtain superior results under linear protocol action recognition. More remarkably, trained larger set YOVO-10M (10M further fine-tuning leads 1.3% gain in top-1 accuracy Kinetics-400 dataset state-of-the-art TAda2D method ImageNet pre-training. Source code available at https://github.com/FuchenUSTC/BCN .